Enhancement of a digital image generally concerns “contrast restoration or amplification” or “image sharpening” within the image.
With sharpening, faint and/or fine details of the image, such as textures (repetition of an elementary pattern), edges and corners of an object are enhanced and made more visible, which would otherwise never have been seen in the original image, making the image appear more crisp and clear.
There exist conventional solutions that sharpen faint/fine details in digital images. They generally operate in the spatial frequency domain where the details are represented by middle (rough details) to high (fine details) frequencies.
More particularly, a controlled amplification of these middle to high frequencies is generally performed, to obtain a controlled improvement of the contrast of the corresponding details. For instance, subtracting the Laplacian function of an image from that image makes the image sharper by enhancing middle to high spatial frequencies (high frequency details).
An operator can access this function through menus provided in image processing software (see Photoshop—commercial brand—which offers for example a de-blurring function). Sometimes, similar functions are embedded in end devices, such as printers or displays.
The main drawback of these conventional solutions lies in the noise increase in these middle to high frequencies.
As explained in the publication WO2005/122086, noise in the image may result from the non-uniform sensitivity of the acquisition device (such as CCD arrays) or medium (such as a photosensitive medium).
The above conventional solutions amplify the acquired noise together with the targeted faint details.
Due to this drawback, the basic image sharpening algorithms are, most often, dedicated to low contrast amplification to prevent any strong amplification of the noise.
As mentioned in the same publication, more sophisticated algorithms combine a de-noising process with a sharpening process.
Such a de-noising process generally assumes that the geometrically closest pixels to a given pixel are the most similar pixels to the given pixel. Based on this assumption, the de-noising process filters a given pixel of the image by averaging the pixels surrounding it.
This is what is called a “local” filtering.
Usually, the latter implements a convolution on each pixel, said convolution being based on a “local” filtering kernel defining a relation between the given pixel and the surrounding closest pixels.
However, removing the noise from textures appears to be a tough task, and most of the time, the de-noising processes also remove small details in the image. Improvement of the image quality is therefore doubtful.
The publication WO2005/122086 and the publication “Image enhancement by non-local reverse heat equation”, A. Buades, B. Coll, J-M. Morel, Preprint CMLA 2006-22 (2006) provides a Non-Local Means (NLM) de-noising algorithm that seeks to overcome the above drawbacks.
The NLM de-noising algorithm uses a filtering kernel per pixel that averages a given pixel with a collection of pixels similar to it. Contrasting with the above “local” filtering kernel, the collection of pixels in an NLM algorithm is distributed within the image without any condition on geometrical distance to the given pixel. This is why the filtering kernel is referred to as “non-local”.
The NLM de-noising algorithm enables efficient de-noising with very good preservation of faint details like textures since the similarity between pixels is taken into account. A better and more efficient enhancement of the original image is therefore obtained.
In more detail, it is known that a digital image is made of an array of samples, for example pixels or voxels, each having one or more component values to represent colours (grayscale value or Red-Green-Blue colour values for example). The NLM de-noising algorithm is applied to each colour component, i.e. once for a grayscale image and three times for an RGB image.
The NLM method comprises the following steps:                obtaining a spatial window, generally a square window but possibly of any other shape that is centred on the particular sample, defining, for a given sample, a patch made of the given sample and samples surrounding the given sample;        computing a plurality of distance values between a first sample and a plurality of other samples to obtain a map of distances, a distance value between two samples representing the similarity between their corresponding patches;        computing, using the map of distances, a plurality of weights corresponding to the plurality of other samples, so as to obtain a map of weights; and        filtering the first sample of the digital image by averaging said samples of the digital image using the map of weights, to obtain an enhanced first sample.        
The NLM algorithm relies on a metric to quantify and compute distances between a given sample (generally a pixel) and each of all or part of all other samples in the image. The metric enables the distance between two samples or pixels to be computed.
For the remainder of the description, reference will be made to pixels. However the invention may be applied to samples other than pixels that form a digital image. A reference to samples may be used herebelow to refer to the general definition of the invention.
As far as the NLM algorithm is concerned, the distance is computed from two pixel values and their respective neighbouring or surrounding pixel values, to quantify a similarity between those two given pixels: two pixels are even more similar when they have the same (colour component) value and, in each of their respective neighbouring pixel pairs (i.e. having the same relative position to the given pixels), the two pixels of the pair also have the same value.
If we name “patch” the set of a pixel and its defined neighbourhood/surrounding of pixels, the distance between two pixels is zero if their respective patches are equal and is large if their respective patches differ greatly.
As an example, a Gaussian weighted Sum of Square Difference (SSD) may be used to compute the “similarity” distance between two pixels.
In the NLM algorithm, a plurality of distances from other pixels of the image is computed for a given first pixel. Generally a distance from each other pixel of the image is computed.
A map of distances or “distance map” comprising all these distances from each other pixel is then built for said given first pixel.
In order to avoid heavy computation, it may be decided to compute such a distance map within a search window centred on the given first pixel.
The distance map then has the same size as the digital image or the search window, as appropriate.
The value in the middle of the distance map of a given first pixel equals ‘zero’ since this “middle” location corresponds to the location of the given first pixel in the image and its value represents the distance between this first pixel and itself.
The other values of the distance map indicate how close, in terms of similarity, the given first pixel and its respective patch are to other pixels and their respective patches.
By using such a distance map, the NLM algorithm seeks to average similar pixels regardless of their geometrical distance from the first pixel, i.e. to average pixels very close in terms of the “similarity” distance as referred to above.
The distance map is then converted into a map of weights, also referred to as “weight map” below, wherein the weighting values are large for similar pixels, and small or zero for dissimilar pixels.
In the above publications, an exponential function is used to convert the distance map. The weight map obtained is made of zeros or positive values.
The given first pixel is subsequently filtered to become a de-noised pixel by using its associated weight map, which operates as a “non-local” filtering kernel. The weight map may be used in a convolution filtering operation, similar to that for the above-mentioned “local” convolution kernel.
This filtering of the first pixel by using the “non-local” filtering kernel computes an average of the pixels that are similar to the given first pixel.
Once all the pixels of the original image have been filtered to become averaged pixels, i.e. de-noised pixels or “enhanced pixels”, an enhanced image made of these enhanced pixels is obtained.
As it may be noted, the NLM algorithm only performs de-noising processing, and then yields a de-noised image as an enhanced image.
In “Image enhancement by non-local reverse heat equation”, an additional enhancing operation is conducted to sharpen the image and make the faint details even more visible.
In this enhancing process, the following operations are performed:    1. computing all the weight maps corresponding to all the pixels of the image to be processed, by implementing the above NLM algorithm;    2. filtering the image by subtracting the Laplacian operator (or “Laplacian”) of the image (made of the Laplacian operator for each pixel of the image) from the image in order to sharpen it, even if the noise spread all over the image is amplified;    3. filtering the sharpened image using the weight maps obtained in step 1. This filtering is similar to the filtering in the NLM algorithm, but is applied to the image modified at step 2; and    4. going back to step 2 as long as one or several enhancement criteria have not been reached.
This process combines multiple iterations of sharpening operations (repetition of step 2 through the loop back from step 4) and of de-noising operations (repetition of step 3 through the same loop back). It results in an improved efficiency of image enhancement of faint details like textures compared to the other known solutions, while keeping a reasonably low level of noise.
However, this solution appears not to be satisfactory enough, in particular because of the noise amplification during the above step 2. Indeed, step 2 still involves striking a balance between improvement of the sharpness and amplification of noise: either the obtained sharpness is low, or the amplified noise is high, requiring a very efficient post de-noising process to be implemented.
Furthermore, the repetition of steps 2 and 3 through the loop back involves a complex processing scheme and considerably increases the processing time.
The present invention aims to solve the above drawbacks.